Date and field issues

[oguzhannysr]

@ZachHoppinen I wanted to make this analysis for Turkey boundary using the code below. But "NoDataInBounds: No data found in bounds." I get the error, and even though I change the dates, it always downloads images between 2020-2021.

from pathlib import Path

# Add main repo to path if you haven't added with conda-develop
# import sys
# sys.path.append('path/to/the/spicy-snow/')

from spicy_snow.retrieval import retrieve_snow_depth
from spicy_snow.IO.user_dates import get_input_dates
import shapely 
# Define the bounding box for Turkey
min_long = 25.0  # Minimum longitude for Turkey
min_lat = 36.0   # Minimum latitude for Turkey
max_long = 45.0  # Maximum longitude for Turkey
max_lat = 42.0   # Maximum latitude for Turkey

# Create a shapely geometry box for Turkey
area = shapely.geometry.box(min_long, min_lat, max_long, max_lat)

# this will be where your results are saved
out_nc = Path('test.nc').expanduser()

# this will generate a tuple of dates from the previous August 1st to this date
dates = get_input_dates("2024-05-01", "2024-02-01") #('2024-04-01') # run on all s1 images from (2020-08-01, 2021-04-01) in this example

#f*64KaD!qfFSiAL
spicy_ds = retrieve_snow_depth(area = area, dates = dates, 
                               work_dir = Path('spicy-snow/').expanduser(), 
                               job_name = f'testing_spicy',
                               existing_job_name = 'testing_spicy',
                               debug=False,
                               outfp=out_nc)

[ZachHoppinen]

Hi @oguzhannysr!

I ran your code and I have a few suggestions for you to start with.

• The way the snow depth retrieval algorithm works it starts from snow free time periods and slow accumulates snow using the S1 backscatter information. Therefore you would want your first s1 images to snow-free. I would suggest running this from some known date that things will be snow free. In North America we generally start on August 1st for this then just be certain that things will snow free for those first images.

• Second, you are running over a huge area! When I ran your code it said it was going to process 852 images. You may want to start with a more focused area to test your code and get a sense of how things are working and looking before you try and run over all of Turkey.

• I am not sure what is happening specifically with the date problem you are having. When I run this code it starts to process images between February and April of 2024 as expected. Maybe you have an old submission called 'testing_spicy'? I would consider changing the job_name and existing_job_name and seeing if that helps. Those are to avoid having you submit jobs over and over again to Hyp3 but if you have already submitted a job in 2020 with that same name it will just assume you want the same data downloaded. So I would start there and let us know if that fixes things.

[oguzhannysr]

Hello, thank you for your advice. I'm trying to change the parts you said. I am continuing to work now. The reason I use this algorithm is that I want to calculate the snow-ice load periodically between each sentinel-1 pair. How consistent do you think this is? For example, my goal is to periodically monitor the snow accumulation in the region I choose every 12 days.

[ZachHoppinen]

I think it depends on the amount of snow accumulation we are talking about. If you are talking about big changes (storms of 0.3-0.5m of new snow) and pretty dry snow I think you have a good chance of this looking reliable. This technique definitely struggles for lower snow amounts (too long a wavelength from Sentinel-1) and for wet snow (though you get a mask for wet snow from it).

Here is a preprint of our analysis of this technique:
https://egusphere.copernicus.org/preprints/2024/egusphere-2024-1018/

Let me know if those changes help. And maybe just print out the dates variable with print(dates) before you run it to be sure that everything looks good there.

[oguzhannysr]

Thank you, when I corrected the parts you said, the code worked but was interrupted.

}
(spicy-snow spicy_snow.utils.spicy_logging INFO) SUBQUERY 1 COMPLETE: results exhausted for subquery
(spicy-snow spicy_snow.utils.spicy_logging INFO) SEARCH COMPLETE: results exhausted for search opts {
    "product_list": [
        "S1A_IW_GRDH_1SDV_20240205T033407_20240205T033432_052418_0656CF_CEAE-GRD_HD"
    ]
}
Combining Sentinel-1 dataArrays: 100%|██████████| 9/9 [00:06<00:00,  1.43it/s]
Downloading IMS snow-cover: 100%|██████████| 9/9 [02:04<00:00, 13.82s/it](spicy-snow spicy_snow.utils.spicy_logging INFO) Downloading fcf.tif.

...78%, 4246 MB, 78 KB/s, 8384 seconds passedTraceback (most recent call last):

  File ~\anaconda3\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File c:\users\yasar\onedrive\masaüstü\personal\personal\scp-code\scipysnow.py:28
    spicy_ds = retrieve_snow_depth(area = area, dates = dates,

  File ~\anaconda3\Lib\site-packages\spicy_snow\retrieval.py:128 in retrieve_snow_depth
    ds = download_fcf(ds, join(work_dir, 'tmp', 'fcf.tif'))

  File ~\anaconda3\Lib\site-packages\spicy_snow\download\forest_cover.py:34 in download_fcf
    url_download(fcf_url, out_fp)

  File ~\anaconda3\Lib\site-packages\spicy_snow\utils\download.py:38 in url_download
    urlretrieve(url, out_fp, reporthook)

  File ~\anaconda3\Lib\urllib\request.py:280 in urlretrieve
    raise ContentTooShortError(

ContentTooShortError: <urlopen error retrieval incomplete: got only 4452628602 out of 5650453608 bytes>

Also, what do the files below mean? Are the values ​​ranging from 0 to 4 in the FCF the value of the snow depth in meters? Are nc files the snow depth value of that date (DoY)?

[oguzhannysr]

...100%, 5388 MB, 100 KB/s, 5436 seconds passed(spicy-snow spicy_snow.utils.spicy_logging INFO) 
(spicy-snow spicy_snow.utils.spicy_logging INFO) Preprocessing Sentinel-1 images
(spicy-snow spicy_snow.utils.spicy_logging INFO) Calculating snow index
(spicy-snow spicy_snow.utils.spicy_logging INFO) Flag wet snow
(spicy-snow spicy_snow.utils.spicy_logging INFO) bottleneck installed. Consider pip uninstalling and re-running if this fails.
(spicy-snow spicy_snow.utils.spicy_logging INFO) bottleneck installed. Consider pip uninstalling and re-running if this fails.
Traceback (most recent call last):

  File ~\anaconda3\Lib\site-packages\xarray\backends\file_manager.py:211 in _acquire_with_cache_info
    file = self._cache[self._key]

  File ~\anaconda3\Lib\site-packages\xarray\backends\lru_cache.py:56 in __getitem__
    value = self._cache[key]

KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('spicy-snow\test.nc',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'ea9d1c60-53fe-42b6-b739-073194e572b5']


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File ~\anaconda3\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File c:\users\yasar\onedrive\masaüstü\personal\personal\scp-code\scipysnow.py:28
    spicy_ds = retrieve_snow_depth(area = area, dates = dates,

  File ~\anaconda3\Lib\site-packages\spicy_snow\retrieval.py:195 in retrieve_snow_depth
    ds.to_netcdf(outfp)

  File ~\anaconda3\Lib\site-packages\xarray\core\dataset.py:1957 in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(

  File ~\anaconda3\Lib\site-packages\xarray\backends\api.py:1255 in to_netcdf
    store = store_open(target, mode, format, group, **kwargs)

  File ~\anaconda3\Lib\site-packages\xarray\backends\netCDF4_.py:391 in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)

  File ~\anaconda3\Lib\site-packages\xarray\backends\netCDF4_.py:338 in __init__
    self.format = self.ds.data_model

  File ~\anaconda3\Lib\site-packages\xarray\backends\netCDF4_.py:400 in ds
    return self._acquire()

  File ~\anaconda3\Lib\site-packages\xarray\backends\netCDF4_.py:394 in _acquire
    with self._manager.acquire_context(needs_lock) as root:

  File ~\anaconda3\Lib\contextlib.py:137 in __enter__
    return next(self.gen)

  File ~\anaconda3\Lib\site-packages\xarray\backends\file_manager.py:199 in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)

  File ~\anaconda3\Lib\site-packages\xarray\backends\file_manager.py:217 in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)

  File src\\netCDF4\\_netCDF4.pyx:2469 in netCDF4._netCDF4.Dataset.__init__

  File src\\netCDF4\\_netCDF4.pyx:2028 in netCDF4._netCDF4._ensure_nc_success

PermissionError: [Errno 13] Permission denied: 'spicy-snow\test.nc'

[ZachHoppinen]

Glad that helped!

The fcf.tif file is the forest cover fraction file. It is quite large and takes a while to download but you should only need to download it once. It looks like you had some server issues while downloading the fcf. I suspect if you remove that file and re try it will successfully download this time. The ims files are snow cover files from an optical satellite platform with 2 meaning no snow and 4 meaning snow. These are used to tell the algorithm when to start integrating snow depths to avoid accumulation snow during unreasonable periods.

[oguzhannysr]

@ZachHoppinen ,Thank you for your help, since the last fcf file was created and no other files were created (except nc files), I still haven't seen a small example of the result.

[oguzhannysr]

Even though I deleted the files and started again, I still got this error.
PermissionError: [Errno 13] Permission denied: 'spicy-snow\test.nc'

[oguzhannysr]

@ZachHoppinen, I got results now, thank you. Can you help me make sense of the results? My .nc result file contains the following variables.

Additionally, snow_index and snow_depth variables were not georeferenced like other layers.

in my nc file:

imranli.zip

[alexiscaro2021]

Even though I deleted the files and started again, I still got this error. PermissionError: [Errno 13] Permission denied: 'spicy-snow\test.nc'

@oguzhannysr
Hi, what did you do? Did you just run the code again? I had the same error, but in my case it takes a long time and each test takes several hours.

[ZachHoppinen]

Glad you got everything to ru. I am not sure why snow_depth wouldn't be georeferenced. They should have the exact same coordinates and geotransformation as the other variables so it should be pretty easy to just make them match another layer's coordinates and geotransform.

Variable description:

• wet_snow: layer showing layers flagged as wet snow (1 = wet, 0 = dry)
• snow_depth: derived snow depth in meters
• ims: snow coverage binary mask (2 = no snow, 4 = snow)
• fcf: forest coverage percentage
• s1: raw sentinel-1 with 3 bands for VV, VH backscatter in dB and incidence angle

All the other layers are intermediate layers for if you want to explore the processing pipeline.

[oguzhannysr]

I still haven't been able to place the snow depth variable in the same coordinate system with the others. What should I do?

[oguzhannysr]

@ZachHoppinen ?

[ZachHoppinen]

What does gdalinfo tell you about the file?

https://gdal.org/programs/gdalinfo.html

[oguzhannysr]

Driver: netCDF/Network Common Data Format
Files: D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc
D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc.aux.xml
Size is 512, 512
Metadata:
NC_GLOBAL#add_offset=0
NC_GLOBAL#AREA_OR_POINT=Point
NC_GLOBAL#bounds={37.90394780805172,39.85085292939551,37.98331048284032,39.90782758019311}
NC_GLOBAL#job_name=imranli2
NC_GLOBAL#param_A=2.5
NC_GLOBAL#param_B=0.2
NC_GLOBAL#param_C=0.55
NC_GLOBAL#resolution=90
NC_GLOBAL#s1_units=dB
NC_GLOBAL#scale_factor=1
NC_GLOBAL#TIFFTAG_DATETIME=2024:05:16 16:19:07
NC_GLOBAL#TIFFTAG_SOFTWARE=Created with GAMMA Software www.gamma-rs.ch data2geotiff v3.0
Subdatasets:
SUBDATASET_1_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":s1
SUBDATASET_1_DESC=[9x3x59x81] s1 (32-bit floating-point)
SUBDATASET_2_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":ims
SUBDATASET_2_DESC=[9x59x81] area_type (32-bit floating-point)
SUBDATASET_3_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":fcf
SUBDATASET_3_DESC=[59x81] fcf (64-bit floating-point)
SUBDATASET_4_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":confidence
SUBDATASET_4_DESC=[59x81] confidence (32-bit floating-point)
SUBDATASET_5_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":deltaCR
SUBDATASET_5_DESC=[9x59x81] deltaCR (32-bit floating-point)
SUBDATASET_6_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":deltaVV
SUBDATASET_6_DESC=[9x59x81] deltaVV (32-bit floating-point)
SUBDATASET_7_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":deltaGamma
SUBDATASET_7_DESC=[59x81x9] deltaGamma (64-bit floating-point)
SUBDATASET_8_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":snow_index
SUBDATASET_8_DESC=[59x81x9] snow_index (64-bit floating-point)
SUBDATASET_9_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":snow_depth
SUBDATASET_9_DESC=[59x81x9] snow_depth (64-bit floating-point)
SUBDATASET_10_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":wet_flag
SUBDATASET_10_DESC=[9x59x81] wet_flag (32-bit floating-point)
SUBDATASET_11_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":alt_wet_flag
SUBDATASET_11_DESC=[9x59x81] alt_wet_flag (32-bit floating-point)
SUBDATASET_12_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":freeze_flag
SUBDATASET_12_DESC=[9x59x81] freeze_flag (32-bit floating-point)
SUBDATASET_13_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":wet_snow
SUBDATASET_13_DESC=[9x59x81] wet_snow (32-bit floating-point)
SUBDATASET_14_NAME=NETCDF:"D:\Data\PROJECTS\CK_ENERJI\METEOROLOJIK\BUZLANMA-KAR\Data\spicy-snow-imranli\imranli.nc":perma_wet
SUBDATASET_14_DESC=[9x59x81] perma_wet (32-bit floating-point)
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 512.0)
Upper Right ( 512.0, 0.0)
Lower Right ( 512.0, 512.0)
Center ( 256.0, 256.0)

[oguzhannysr]

I sent the NC file above, maybe it will be better understood if you can also review it.

[ZachHoppinen]

I don't know why your corner coordinates don't correspond to EPSG:4326. Maybe try it in a different mapping software? Or try a gdal_translate to explicitely set the data into the right EPSG?