Merge branch 'develop' into feature/xr_dataset_context_manager #994
GitHub Actions / Core / Unit Test Results (3.9)
failed
Oct 3, 2024 in 0s
23 fail, 676 pass in 7m 38s
Annotations
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_additional_variables (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1fb2dfd0>
fp = <http.client.HTTPResponse object at 0x7f0e1e558460>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1e55af40>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2017242N16333', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = ['numobs', 'season', 'number', 'subbasin', 'name', 'source_usa', ...]
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_additional_variables>
def test_ibtracs_additional_variables(self):
"""Check additional_variables option"""
# this is the complete list (as of May 2023) of variables in IBTrACS that are not
# agency-specific and that are not already considered by other parts of
# `from_ibtracs_netcdf`:
addtl_vars = [
'numobs', 'season', 'number', 'subbasin', 'name', 'source_usa', 'source_jma',
'source_cma', 'source_hko', 'source_new', 'source_reu', 'source_bom', 'source_nad',
'source_wel', 'source_td5', 'source_td6', 'source_ds8', 'source_neu', 'source_mlc',
'iso_time', 'nature', 'wmo_wind', 'wmo_pres', 'wmo_agency', 'track_type',
'main_track_sid', 'dist2land', 'landfall', 'iflag', 'storm_speed', 'storm_dir',
]
> tc_track = tc.TCTracks.from_ibtracs_netcdf(
storm_id='2017242N16333',
additional_variables=addtl_vars,
)
climada/hazard/test/test_tc_tracks.py:272:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2017242N16333', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = ['numobs', 'season', 'number', 'subbasin', 'name', 'source_usa', ...]
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_antimeridian (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1fd22be0>
fp = <http.client.HTTPResponse object at 0x7f0e1fd22490>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1fd22580>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = ['official_3h']
rescale_windspeeds = True, storm_id = '2013224N12220', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_antimeridian>
def test_ibtracs_antimeridian(self):
"""Read a track that crosses the antimeridian and make sure that lon is consistent"""
storm_id = '2013224N12220'
# the officially responsible agencies 'usa' and 'tokyo' use different signs in lon, but we
# have to `estimate_missing` because both have gaps in reported values
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id=storm_id, provider=['official_3h'],
estimate_missing=True)
climada/hazard/test/test_tc_tracks.py:152:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = ['official_3h']
rescale_windspeeds = True, storm_id = '2013224N12220', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_correct_pass (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e25ad16d0>
fp = <http.client.HTTPResponse object at 0x7f0e74cc29d0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e25448460>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1982267N25289', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_correct_pass>
def test_ibtracs_correct_pass(self):
"""Check estimate_missing option"""
> tc_try = tc.TCTracks.from_ibtracs_netcdf(
provider='usa', storm_id='1982267N25289', estimate_missing=True)
climada/hazard/test/test_tc_tracks.py:240:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1982267N25289', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_discard_single_points (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1f908430>
fp = <http.client.HTTPResponse object at 0x7f0e1f908970>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1f908910>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = None, year_range = (1863, 1863)
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = False
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_discard_single_points>
def test_ibtracs_discard_single_points(self):
"""Check discard_single_points option"""
passed = False
for year in range(1863, 1981):
> tc_track_singlept = tc.TCTracks.from_ibtracs_netcdf(
provider='usa', year_range=(year,year), discard_single_points=False)
climada/hazard/test/test_tc_tracks.py:250:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = None, year_range = (1863, 1863)
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = False
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_estimate_missing (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e2616f910>
fp = <http.client.HTTPResponse object at 0x7f0e2616f550>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e2616f5e0>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2012152N12130', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_estimate_missing>
def test_ibtracs_estimate_missing(self):
"""Read a tropical cyclone and estimate missing values."""
storm_id = '2012152N12130'
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id=storm_id, estimate_missing=True)
climada/hazard/test/test_tc_tracks.py:161:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2012152N12130', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_interpolate_missing (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1fcad250>
fp = <http.client.HTTPResponse object at 0x7f0e1fcad6a0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1fcad790>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2010066S19050', year_range = None
basin = None, genesis_basin = None, interpolate_missing = False
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_interpolate_missing>
def test_ibtracs_interpolate_missing(self):
"""Read a tropical cyclone with and without interpolating missing values."""
storm_id = '2010066S19050'
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id=storm_id, interpolate_missing=False)
climada/hazard/test/test_tc_tracks.py:208:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2010066S19050', year_range = None
basin = None, genesis_basin = None, interpolate_missing = False
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_official (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e254480a0>
fp = <http.client.HTTPResponse object at 0x7f0e25448280>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1ef7efd0>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'official'
rescale_windspeeds = True, storm_id = '2012152N12130', year_range = None
basin = None, genesis_basin = None, interpolate_missing = False
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_official>
def test_ibtracs_official(self):
"""Read a tropical cyclone, only officially reported values."""
storm_id = '2012152N12130'
> tc_track = tc.TCTracks.from_ibtracs_netcdf(
storm_id=storm_id, interpolate_missing=False, provider='official')
climada/hazard/test/test_tc_tracks.py:183:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'official'
rescale_windspeeds = True, storm_id = '2012152N12130', year_range = None
basin = None, genesis_basin = None, interpolate_missing = False
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_range (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1f47a3d0>
fp = <http.client.HTTPResponse object at 0x7f0e1f47a100>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1f47a640>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = None, year_range = (2100, 2150)
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_range>
def test_ibtracs_range(self):
"""Read several TCs."""
> tc_track = tc.TCTracks.from_ibtracs_netcdf(year_range=(2100, 2150))
climada/hazard/test/test_tc_tracks.py:223:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = None, year_range = (2100, 2150)
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_raw_pass (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e254d7460>
fp = <http.client.HTTPResponse object at 0x7f0e254d79a0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e74cc2340>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2017242N16333', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_raw_pass>
def test_ibtracs_raw_pass(self):
"""Read a tropical cyclone."""
# read without specified provider or estimation of missing values
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id='2017242N16333')
climada/hazard/test/test_tc_tracks.py:86:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2017242N16333', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_scale_wind (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1fc98580>
fp = <http.client.HTTPResponse object at 0x7f0e1fc98be0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1fc98340>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2012152N12130', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_scale_wind>
def test_ibtracs_scale_wind(self):
"""Read a tropical cyclone and scale wind speed according to agency."""
storm_id = '2012152N12130'
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id=storm_id, rescale_windspeeds=True)
climada/hazard/test/test_tc_tracks.py:196:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = '2012152N12130', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_ibtracs_with_provider (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1f343e20>
fp = <http.client.HTTPResponse object at 0x7f0e1f343a00>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1f343820>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '2012152N12130', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_ibtracs_with_provider>
def test_ibtracs_with_provider(self):
"""Read a tropical cyclone with and without explicit provider."""
storm_id = '2012152N12130'
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id=storm_id, provider='usa')
climada/hazard/test/test_tc_tracks.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '2012152N12130', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_penv_rmax_penv_pass (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1eabc280>
fp = <http.client.HTTPResponse object at 0x7f0e1eabccd0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1f47a5b0>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1992230N11325', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_penv_rmax_penv_pass>
def test_penv_rmax_penv_pass(self):
"""from_ibtracs_netcdf"""
> tc_track = tc.TCTracks.from_ibtracs_netcdf(provider='usa', storm_id='1992230N11325')
climada/hazard/test/test_tc_tracks.py:74:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1992230N11325', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_raw_ibtracs_empty_pass (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1f45c5b0>
fp = <http.client.HTTPResponse object at 0x7f0e1f45c1f0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1f45c400>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1988234N13299', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_raw_ibtracs_empty_pass>
def test_raw_ibtracs_empty_pass(self):
"""Test reading empty TC from IBTrACS files"""
> tc_track = tc.TCTracks.from_ibtracs_netcdf(
provider='usa', storm_id='1988234N13299')
climada/hazard/test/test_tc_tracks.py:55:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1988234N13299', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIbtracs
github-actions / Core / Unit Test Results (3.9)
test_raw_ibtracs_invalid_pass (climada.hazard.test.test_tc_tracks.TestIbtracs) failed
tests_xml/tests.xml [took 0s]
Raw output
AssertionError: 'IDs are invalid' not found in 'Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc'
self = <climada.hazard.test.test_tc_tracks.TestIbtracs testMethod=test_raw_ibtracs_invalid_pass>
def test_raw_ibtracs_invalid_pass(self):
"""Test reading invalid/non-existing TC from IBTrACS files"""
with self.assertRaises(ValueError) as cm:
tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id='INVALID')
> self.assertIn("IDs are invalid", str(cm.exception))
E AssertionError: 'IDs are invalid' not found in 'Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc'
climada/hazard/test/test_tc_tracks.py:64: AssertionError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIO
github-actions / Core / Unit Test Results (3.9)
test_hdf5_io (climada.hazard.test.test_tc_tracks.TestIO) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1ee66cd0>
fp = <http.client.HTTPResponse object at 0x7f0e1ee66490>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e25444670>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = None, year_range = (1993, 1994)
basin = 'EP', genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIO testMethod=test_hdf5_io>
def test_hdf5_io(self):
"""Test writing and reading hdf5 TCTracks instances"""
path = DATA_DIR.joinpath("tc_tracks.h5")
> tc_track = tc.TCTracks.from_ibtracs_netcdf(
provider='usa', year_range=(1993, 1994), basin='EP', estimate_missing=True)
climada/hazard/test/test_tc_tracks.py:325:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = None, year_range = (1993, 1994)
basin = 'EP', genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestIO
github-actions / Core / Unit Test Results (3.9)
test_netcdf_io (climada.hazard.test.test_tc_tracks.TestIO) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1faa9dc0>
fp = <http.client.HTTPResponse object at 0x7f0e1faa9b50>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1faa9730>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1988234N13299', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = ['numobs', 'storm_speed', 'nature']
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestIO testMethod=test_netcdf_io>
def test_netcdf_io(self):
"""Test writting and reading netcdf4 TCTracks instances"""
path = DATA_DIR.joinpath("tc_tracks_nc")
path.mkdir(exist_ok=True)
> tc_track = tc.TCTracks.from_ibtracs_netcdf(
provider='usa', storm_id='1988234N13299', estimate_missing=True,
additional_variables=["numobs", "storm_speed", "nature"],
)
climada/hazard/test/test_tc_tracks.py:297:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1988234N13299', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = True, correct_pres = False, discard_single_points = True
additional_variables = ['numobs', 'storm_speed', 'nature']
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestFuncs
github-actions / Core / Unit Test Results (3.9)
test_generate_centroids (climada.hazard.test.test_tc_tracks.TestFuncs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e25472760>
fp = <http.client.HTTPResponse object at 0x7f0e254488e0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e25448760>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = ['usa', 'bom']
rescale_windspeeds = True
storm_id = ['1988169N14259', '2002073S16161', '2002143S07157']
year_range = None, basin = None, genesis_basin = None
interpolate_missing = True, estimate_missing = False, correct_pres = False
discard_single_points = True, additional_variables = None
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestFuncs testMethod=test_generate_centroids>
def test_generate_centroids(self):
"""Test centroids generation feature."""
storms = ['1988169N14259', '2002073S16161', '2002143S07157']
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id=storms, provider=["usa", "bom"])
climada/hazard/test/test_tc_tracks.py:589:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = ['usa', 'bom']
rescale_windspeeds = True
storm_id = ['1988169N14259', '2002073S16161', '2002143S07157']
year_range = None, basin = None, genesis_basin = None
interpolate_missing = True, estimate_missing = False, correct_pres = False
discard_single_points = True, additional_variables = None
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestFuncs
github-actions / Core / Unit Test Results (3.9)
test_get_extent (climada.hazard.test.test_tc_tracks.TestFuncs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1fd1df70>
fp = <http.client.HTTPResponse object at 0x7f0e1fd1d370>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1fd1d670>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = ['usa', 'bom']
rescale_windspeeds = True
storm_id = ['1988169N14259', '2002073S16161', '2002143S07157']
year_range = None, basin = None, genesis_basin = None
interpolate_missing = True, estimate_missing = False, correct_pres = False
discard_single_points = True, additional_variables = None
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestFuncs testMethod=test_get_extent>
def test_get_extent(self):
"""Test extent/bounds attributes."""
storms = ['1988169N14259', '2002073S16161', '2002143S07157']
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id=storms, provider=["usa", "bom"])
climada/hazard/test/test_tc_tracks.py:579:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = ['usa', 'bom']
rescale_windspeeds = True
storm_id = ['1988169N14259', '2002073S16161', '2002143S07157']
year_range = None, basin = None, genesis_basin = None
interpolate_missing = True, estimate_missing = False, correct_pres = False
discard_single_points = True, additional_variables = None
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestFuncs
github-actions / Core / Unit Test Results (3.9)
test_subset (climada.hazard.test.test_tc_tracks.TestFuncs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1f4aa9d0>
fp = <http.client.HTTPResponse object at 0x7f0e1ee45af0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1ee45580>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True
storm_id = ['1988169N14259', '2002073S16161', '2002143S07157']
year_range = None, basin = None, genesis_basin = None
interpolate_missing = True, estimate_missing = False, correct_pres = False
discard_single_points = True, additional_variables = None
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestFuncs testMethod=test_subset>
def test_subset(self):
"""Test subset."""
storms = ['1988169N14259', '2002073S16161', '2002143S07157']
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id=storms)
climada/hazard/test/test_tc_tracks.py:573:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True
storm_id = ['1988169N14259', '2002073S16161', '2002143S07157']
year_range = None, basin = None, genesis_basin = None
interpolate_missing = True, estimate_missing = False, correct_pres = False
discard_single_points = True, additional_variables = None
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks.TestFuncs
github-actions / Core / Unit Test Results (3.9)
test_tracks_in_exp_pass (climada.hazard.test.test_tc_tracks.TestFuncs) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1f495e80>
fp = <http.client.HTTPResponse object at 0x7f0e1f557e20>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1f557f40>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = ['2000233N12316', '2000160N21267']
year_range = None, basin = None, genesis_basin = None
interpolate_missing = True, estimate_missing = False, correct_pres = False
discard_single_points = True, additional_variables = None
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks.TestFuncs testMethod=test_tracks_in_exp_pass>
def test_tracks_in_exp_pass(self):
"""Check if tracks in exp are filtered correctly"""
# Load two tracks from ibtracks
storms = {'in': '2000233N12316', 'out': '2000160N21267'}
> tc_track = tc.TCTracks.from_ibtracs_netcdf(storm_id=list(storms.values()))
climada/hazard/test/test_tc_tracks.py:898:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = ['2000233N12316', '2000160N21267']
year_range = None, basin = None, genesis_basin = None
interpolate_missing = True, estimate_missing = False, correct_pres = False
discard_single_points = True, additional_variables = None
file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks_synth.TestDecay
github-actions / Core / Unit Test Results (3.9)
test_wrong_decay_pass (climada.hazard.test.test_tc_tracks_synth.TestDecay) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1f45c700>
fp = <http.client.HTTPResponse object at 0x7f0e27a08ac0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1f82ac40>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1975178N28281', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks_synth.TestDecay testMethod=test_wrong_decay_pass>
def test_wrong_decay_pass(self):
"""Test decay not implemented when coefficient < 1"""
> track = tc.TCTracks.from_ibtracs_netcdf(provider='usa', storm_id='1975178N28281')
climada/hazard/test/test_tc_tracks_synth.py:293:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = '1975178N28281', year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_tc_tracks_synth.TestSynth
github-actions / Core / Unit Test Results (3.9)
test_random_walk_single_point (climada.hazard.test.test_tc_tracks_synth.TestSynth) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1e3bd580>
fp = <http.client.HTTPResponse object at 0x7f0e1e397580>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1e397d30>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = None, year_range = (1951, 1951)
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = False
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_tc_tracks_synth.TestSynth testMethod=test_random_walk_single_point>
def test_random_walk_single_point(self):
found = False
for year in range(1951, 1981):
> tc_track = tc.TCTracks.from_ibtracs_netcdf(provider='usa',
year_range=(year,year),
discard_single_points=False)
climada/hazard/test/test_tc_tracks_synth.py:586:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = 'usa'
rescale_windspeeds = True, storm_id = None, year_range = (1951, 1951)
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = False
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Check warning on line 0 in climada.hazard.test.test_trop_cyclone.TestReader
github-actions / Core / Unit Test Results (3.9)
test_cross_antimeridian (climada.hazard.test.test_trop_cyclone.TestReader) failed
tests_xml/tests.xml [took 0s]
Raw output
ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
> urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
climada/util/files_handler.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:239: in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:214: in urlopen
return opener.open(url, data, timeout)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:523: in open
response = meth(req, response)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:632: in http_response
response = self.parent.error(
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:561: in error
return self._call_chain(*args)
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:494: in _call_chain
result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <urllib.request.HTTPDefaultErrorHandler object at 0x7f0e27570460>
req = <urllib.request.Request object at 0x7f0e1e295bb0>
fp = <http.client.HTTPResponse object at 0x7f0e1e2950d0>, code = 404
msg = 'Not Found', hdrs = <http.client.HTTPMessage object at 0x7f0e1e1c0250>
def http_error_default(self, req, fp, code, msg, hdrs):
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
E urllib.error.HTTPError: HTTP Error 404: Not Found
../../../micromamba/envs/climada_env_3.9/lib/python3.9/urllib/request.py:641: HTTPError
The above exception was the direct cause of the following exception:
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = ['2020346S13168'], year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
> download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
climada/hazard/tc_tracks.py:459:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
url = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc'
file_name = 'IBTrACS.ALL.v04r00.nc'
def download_ftp(url, file_name):
"""Download file from ftp in current folder.
Parameters
----------
url : str
url containing data to download
file_name : str
name of the file to dowload
Raises
------
ValueError
"""
LOGGER.info('Downloading file %s', file_name)
try:
with DownloadProgressBar(unit='B', unit_scale=True, miniters=1,
desc=url.split('/')[-1]) as prog_bar:
urllib.request.urlretrieve(url, file_name, reporthook=prog_bar.update_to)
except Exception as exc:
> raise ValueError(
f'{exc.__class__} - "{exc}": failed to retrieve {url} into {file_name}'
) from exc
E ValueError: <class 'urllib.error.HTTPError'> - "HTTP Error 404: Not Found": failed to retrieve https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf/IBTrACS.ALL.v04r00.nc into IBTrACS.ALL.v04r00.nc
climada/util/files_handler.py:131: ValueError
The above exception was the direct cause of the following exception:
self = <climada.hazard.test.test_trop_cyclone.TestReader testMethod=test_cross_antimeridian>
def test_cross_antimeridian(self):
# Two locations on the island Taveuni (Fiji), one west and one east of 180° longitude.
# We list the second point twice, with different lon-normalization:
cen = Centroids.from_lat_lon([-16.95, -16.8, -16.8], [179.9, 180.1, -179.9])
cen.set_dist_coast(precomputed=True)
# Cyclone YASA (2020) passed directly over Fiji
> tr = TCTracks.from_ibtracs_netcdf(storm_id=["2020346S13168"])
climada/hazard/test/test_trop_cyclone.py:134:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'climada.hazard.tc_tracks.TCTracks'>, provider = None
rescale_windspeeds = True, storm_id = ['2020346S13168'], year_range = None
basin = None, genesis_basin = None, interpolate_missing = True
estimate_missing = False, correct_pres = False, discard_single_points = True
additional_variables = None, file_name = 'IBTrACS.ALL.v04r00.nc'
@classmethod
def from_ibtracs_netcdf(cls, provider=None, rescale_windspeeds=True, storm_id=None,
year_range=None, basin=None, genesis_basin=None,
interpolate_missing=True, estimate_missing=False, correct_pres=False,
discard_single_points=True, additional_variables=None,
file_name='IBTrACS.ALL.v04r00.nc'):
"""Create new TCTracks object from IBTrACS databse.
When using data from IBTrACS, make sure to be familiar with the scope and limitations of
IBTrACS, e.g. by reading the official documentation
(https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf).
Reading the CLIMADA documentation can't replace a thorough understanding of the underlying
data. This function only provides a (hopefully useful) interface for the data input, but
cannot provide any guidance or make recommendations about if and how to use IBTrACS data
for your particular project.
Resulting tracks are required to have both pressure and wind speed information at all time
steps. Therefore, all track positions where one of wind speed or pressure are missing are
discarded unless one of `interpolate_missing` or `estimate_missing` are active.
Some corrections are automatically applied, such as: `environmental_pressure` is enforced
to be larger than `central_pressure`.
Note that the tracks returned by this function might contain irregular time steps since
that is often the case for the original IBTrACS records: many agencies add an additional
time step at landfall. Apply the `equal_timestep` function afterwards to enforce regular
time steps.
Parameters
----------
provider : str or list of str, optional
Either specify an agency, such as "usa", "newdelhi", "bom", "cma", "tokyo", or the
special values "official" and "official_3h":
* "official" means using the (usually 6-hourly) officially reported values of the
officially responsible agencies.
* "official_3h" means to include (inofficial) 3-hourly data of the officially
responsible agencies (whenever available).
If you want to restrict to the officially reported values by the officially responsible
agencies (`provider="official"`) without any modifications to the original official
data, make sure to also set `estimate_missing=False` and `interpolate_missing=False`.
Otherwise, gaps in the official reporting will be filled using interpolation and/or
statistical estimation procedures (see below).
If a list is given, the following logic is applied: For each storm, the variables that
are not reported by the first agency for this storm are taken from the next agency in
the list that did report this variable for this storm. For different storms, the same
variable might be taken from different agencies.
Default: ``['official_3h', 'usa', 'tokyo', 'newdelhi', 'reunion', 'bom', 'nadi',
'wellington', 'cma', 'hko', 'ds824', 'td9636', 'td9635', 'neumann', 'mlc']``
rescale_windspeeds : bool, optional
If True, all wind speeds are linearly rescaled to 1-minute sustained winds.
Note however that the IBTrACS documentation (Section 5.2,
https://www.ncei.noaa.gov/sites/default/files/2021-07/IBTrACS_version4_Technical_Details.pdf)
includes a warning about this kind of conversion: "While a multiplicative factor can
the numerical differences, there are procedural and observational differences between
agencies that can change through time, which confounds the simple multiplicative
factor." Default: True
storm_id : str or list of str, optional
IBTrACS ID of the storm, e.g. 1988234N13299, [1988234N13299, 1989260N11316].
year_range : tuple (min_year, max_year), optional
Year range to filter track selection. Default: None.
basin : str, optional
If given, select storms that have at least one position in the specified basin. This
allows analysis of a given basin, but also means that basin-specific track sets should
not be combined across basins since some storms will be in more than one set. If you
would like to select storms by their (unique) genesis basin instead, use the parameter
`genesis_basin`. For possible values (basin abbreviations), see the parameter
`genesis_basin`. If None, this filter is not applied. Default: None.
genesis_basin : str, optional
The basin where a TC is formed is not defined in IBTrACS. However, this filter option
allows to restrict to storms whose first valid eye position is in the specified basin,
which simulates the genesis location. Note that the resulting genesis basin of a
particular track may depend on the selected `provider` and on `estimate_missing`
because only the first *valid* eye position is considered. Possible values are 'NA'
(North Atlantic), 'SA' (South Atlantic), 'EP' (Eastern North Pacific, which includes
the Central Pacific region), 'WP' (Western North Pacific), 'SP' (South Pacific),
'SI' (South Indian), 'NI' (North Indian). If None, this filter is not applied.
Default: None.
interpolate_missing : bool, optional
If True, interpolate temporal reporting gaps within a variable (such as pressure, wind
speed, or radius) linearly if possible. Temporal interpolation is with respect to the
time steps defined in IBTrACS for a particular storm. No new time steps are added that
are not originally defined in IBTrACS.
For each time step with a missing value, this procedure is only able to fill in that
value if there are other time steps before and after this time step for which values
have been reported.
This procedure will be applied before the statistical estimations referred to
by `estimate_missing`. It is applied to all variables (eye position, wind speed,
environmental and central pressure, storm radius and radius of maximum winds).
Default: True
estimate_missing : bool, optional
For each fixed time step, estimate missing pressure, wind speed and radius using other
variables that are available at that time step.
The relationships between the variables are purely statistical. In comparison to
`interpolate_missing`, this procedure is able to estimate values for variables that
haven't been reported by any agency at any time step, as long as other variables are
available.
A typical example are storms before 1950, for which there are often no reported values
for pressure, but for wind speed. In this case, a rough statistical pressure-wind
relationship is applied to estimate the missing pressure values from the available
wind-speed values.
Make sure to set `rescale_windspeeds=True` when using this option because the
statistical relationships are calibrated using rescaled wind speeds.
Default: False
correct_pres : bool, optional
For backwards compatibility, alias for `estimate_missing`.
This is deprecated, use `estimate_missing` instead!
discard_single_points : bool, optional
Whether to discard tracks that consists of a single point. Recommended for full
compatiblity with other functions such as `equal_timesteps`. Default: True.
file_name : str, optional
Name of NetCDF file to be dowloaded or located at climada/data/system.
Default: 'IBTrACS.ALL.v04r00.nc'
additional_variables : list of str, optional
If specified, additional IBTrACS data variables are extracted, such as "nature" or
"storm_speed". Only variables that are not agency-specific are supported.
Default: None.
Returns
-------
tracks : TCTracks
TCTracks with data from IBTrACS
"""
if correct_pres:
LOGGER.warning("`correct_pres` is deprecated. "
"Use `estimate_missing` instead.")
estimate_missing = True
if estimate_missing and not rescale_windspeeds:
LOGGER.warning(
"Using `estimate_missing` without `rescale_windspeeds` is strongly discouraged!")
ibtracs_path = SYSTEM_DIR.joinpath(file_name)
if not ibtracs_path.is_file():
try:
download_ftp(f'{IBTRACS_URL}/{IBTRACS_FILE}', IBTRACS_FILE)
shutil.move(IBTRACS_FILE, ibtracs_path)
except ValueError as err:
> raise ValueError(
f'Error while downloading {IBTRACS_URL}. Try to download it manually and '
f'put the file in {ibtracs_path}') from err
E ValueError: Error while downloading https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r00/access/netcdf. Try to download it manually and put the file in /home/runner/climada/data/IBTrACS.ALL.v04r00.nc
climada/hazard/tc_tracks.py:462: ValueError
Loading