Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add error handling for faulty downloads #20

Open
fernandolis10 opened this issue Mar 3, 2021 · 2 comments
Open

Add error handling for faulty downloads #20

fernandolis10 opened this issue Mar 3, 2021 · 2 comments

Comments

@fernandolis10
Copy link
Collaborator

fernandolis10 commented Mar 3, 2021

For all commands that depend on the original image being downloaded properly (e.g. tiling, writing to tfrecords, generating video), implement some sort of check that the original images we are using are what we expect them to be. Things to consider:

  • check that image size is as expected
  • check that pixel values are as expected (not completely missing from image)

Alternatively, instead of implementing the check while tilling or writing to tfrecords AFTER the images are downloaded, catch any errors from the call to gdal_translate which may signal that the data being downloaded is faulty WHILE it is downloading.

Unexpected errors/warnings that have sometimes occurred in downloading:

  • ERROR 1: GDALWMS: Unable to download block 96, 0. URL: Failed to connect to gibs.earthdata.nasa.gov port 443: Timed out. HTTP status code: 0, error: Failed to connect to gibs.earthdata.nasa.gov port 443: Timed out.
  • Warning 1: libjpeg: Premature end of JPEG file (this warning can be turned as an error by setting GDAL_ERROR_ON_LIBJPEG_WARNING to TRUE)
@mcechini
Copy link
Collaborator

mcechini commented Mar 3, 2021

I think we'd be interested in knowing what requests failed and when the occurred. That way we can see if there is a server/service issue on our end.

@fernandolis10
Copy link
Collaborator Author

I think we'd be interested in knowing what requests failed and when the occurred. That way we can see if there is a server/service issue on our end.

There was a request today (3/3/21) for an image of the whole world, which should have occurred between 10am-12:30pm est which resulted in the timeout error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants