clean up and final push

Author: behzad-vahedi

README.md

@@ -11,10 +11,12 @@ Project funded by NASA Applied Sciences Program and developed by ESOC and Earth
 Our project aims to extende the EarthLab’s "Fire Event Delineation for Python (FIREDpy)" library by updating it for Near-Real-Time (NRT) fire event perimeter mapping through the fusion of optical and radar remote sensing data. Our main motivations are twofold: to achieve closer-to-real-time temporal resolution and to improve the spatial resolution of the FIREDpy outputs. 
 
 In the first stage of the project, we focused on answering two questions:
-- Is applying BRDF correction necessary to generated NRT fire perimeters?
-- How does fire affect the signal in Sentinel-2 and Landsat-8 images and how do these satellites compare?
+- __Q1.__ Is applying BRDF correction necessary to generated NRT fire perimeters?
+- __Q2.__ How does fire affect the signal in Sentinel-2 and Landsat-8 images and how do these satellites compare?
 
-### BERDF Correction
+Below, I will explain how I approached each question. To run the scripts in this repository, you should create a conda environment using the `environment.yml` file. Follow the instructions provided [here](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file) to do so.
+
+### Q1. BERDF Correction
 The flowchart below demonstrates the steps I took to answer the first question, with the caveat that I only tested BRDF on Sentinel-2 images (using the Sen2nbar package) since I couldn't find any python implementation of this correction for Landsat images.
 
 <img src="./images/Flowchart.jpg" width=800>
@@ -33,22 +35,36 @@ or alternatively,
 
 In these commands, the value of the `event_id` (or `id`) argument must be the ID of the fire event for which satellite images are to be searched. This comes from the id field of the [fire events shapefile](./fire_events/Fire_events.zip). The value of the `satellite` (or `s`) argument must be the name of the satellite, either "sentinel" or "landsat".
 
+#### Important Note:
+Before running the `optical_scenes.py` script, you should create separate accounts for Landsat and Sentinel API and set your username and password in the corresponidng variables in the `config.py` file.
+
 You can set or change the search criteria in the `config.py` file. These parameters are:
 - `max_cloud_cover`: Maximum cloud cover percentage. default: 40%.
 - `delta_days_landsat`: The number of days to use as a buffer before and after the fire event when searching for Landsat images. default: 70 days.
 - `delta_days_sentinel`: The number of days to use as a buffer before and after the fire event when searching for Sentinel-2 images. default: 40 days.
-- `download_scenes`: Whether to download images or not. default: False. __CAUTION: depending on the number of the images found, this could take a long time and might reqiure a huge amount of storage__
-- `save_footprints`: Whether to save image footprints or not. default: False.
+- `download_scenes`: Whether to download images or not. default: False. __CAUTION: depending on the number of the images found, this could take a long time and might reqiure a huge amount of storage__.
+- `save_footprints`: Whether to save image footprints or not. default: True.
 - `update_json`: deprecated.
 - `data_dir`: The directory where the images and/or their footprints will be saved. See below for more information.
 
+My suggestion is to first run the script with the default values for `download_scenes` and `save_footprints`. This will give you the footprints and all the available metadata of all the images that satisfy the search criteria in a geojson format. You can then look at the image footprint and metadata, decide which images you'd like to download, and then run the script again to download only those images (with `download_scenes=True`). However, if you decide to do so, __you should remember to modify the `get_landsat()` and `get_sentinel()` functions inside `optical_scenes.py` following the instructions provided there__
+
 #### Directory Organization
-`data_dir` is the directory where the you would want the images to be saved. Once you create this directory, extract (unzip) the `Fire_events.zip` there. This will (should) create a new subdirectory called Fire_events. Per each fire id you run the `optical_scenes.py` script for, a new subdirectory will be created within "Fire_events". Also, depending on what the `-s` parameter is, the corresponding subdirectory will be created within the fire_id folder where the footprints and images will be saved.
+`data_dir` is the directory where the you would want the images to be saved. Once you create this directory, extract (unzip) the "Fire_events.zip" there. This will (should) create a new subdirectory called Fire_events. Per each fire id you run the `optical_scenes.py` script for, a new subdirectory will be created within "Fire_events". Also, depending on what the `-s` parameter is, the corresponding subdirectory will be created within the fire_id folder where the footprints and images will be saved.
 
 <img src="./images/directory_tree.png" width="300">
 
 
-#### Important Note:
-Before running the `optical_scenes.py` script, you should create separate accounts for Landsat and Sentinel API and set your username and password in the corresponidng variables in the `config.py` file.
 
-### Optical Signal Evaluation
+### Q2. Optical Signal Evaluation
+To perform the optical signal evaluation and visualize the results, you can use the jupyter notebooks provided here: One for Sentinel-2 ([Sentinel Image Analysis.ipynb](./scripts/Sentinel%20Image%20Analysis.ipynb)) images, and one for Landsat-8 images ([Landsat Image Analysis.ipynb](./scripts/Landsat%20Image%20Analysis.ipynb)). These notebooks are designed in a modular fashion, meaning that the functions used in these notebooks can be used outside of the notebooks for further investivation and analysis. They should be self-explanatory given the docstrings and comments provided. If you have any questions, please contact Behzad.
+
+Assuming that you have already downlaoded the pre- and post-fire images for that fire in the previous step, to run these notebooks, you should: 
+1. Change `fire_id` in cell #3  to the fire ID of your interest
+2. Change `image_dir` in cell #4 to the directory where your images are located.
+3. Change `fire_events` to reflect the directory where your fire shapefile is saved (assuming you have followed the directory structure explained above).
+4. The `MASK_IMAGES` variable in these notebooks determines whether the calculations should be masked to the perimeter of the fire or not. Recommended values: `True` when calculating dNBR or applying BRDF, `False` when you want to create and RGB or false color composite image.
+5. These notebooks assume _exactly one_ image before the fire and _exactly one_ image after the fire. If you have more images, you should modify the notebooks.
+
+You should run the notebooks once per fire event.
+

environment.yml

@@ -17,16 +17,8 @@
     "from glob import glob\n",
     "import os\n",
     "from matplotlib.colors import ListedColormap, BoundaryNorm\n",
-    "from utils import draw_legend"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "bb4b746e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
+    "from utils import draw_legend\n",
+    "from config import Config\n",
     "from rasterio.enums import Resampling"
    ]
   },
@@ -38,7 +30,9 @@
    "outputs": [],
    "source": [
     "import warnings\n",
-    "warnings.filterwarnings('ignore')"
+    "warnings.filterwarnings('ignore')\n",
+    "config = Config()\n",
+    "dir_separator = config.dir_sep"
    ]
   },
   {
@@ -51,6 +45,17 @@
     "fire_id = \"202\""
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "66357e9e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "image_dir = f\"E:/Projects/UCB/FiredPy/data/Fire_events/{fire_id}/Landsat/\"\n",
+    "file_address = glob(os.path.join(image_dir, '*_T1'))"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 5,
@@ -145,7 +150,7 @@
     "        xarray.DataArray: The multi-band image with the NIR and SWIR bands.\n",
     "    \"\"\"\n",
     "    bands = []    \n",
-    "    image_path = image_dir.split(\"\\\\\")[-1]\n",
+    "    image_path = image_dir.split(dir_separator)[-1]\n",
     "    \n",
     "    if MASK_IMAGES:\n",
     "        nir = rioxarray.open_rasterio(f\"{image_dir}/{image_path}_B5.{ext}\").rio.clip(mask_geom, fire_events.crs)\n",
@@ -205,7 +210,7 @@
     "    Returns:\n",
     "        xarray.DataArray: The RGB image with the red, green, and blue bands.\n",
     "    \"\"\"\n",
-    "    image_path = image_dir.split(\"\\\\\")[-1]\n",
+    "    image_path = image_dir.split(dir_separator)[-1]\n",
     "    \n",
     "    if MASK_IMAGES:\n",
     "        rgb = [rioxarray.open_rasterio(f\"{image_dir}/{image_path}_B{x}.{ext}\").rio.clip(mask_geom, fire_events.crs) for x in [4,3,2]]\n",
@@ -249,7 +254,7 @@
     "    Returns:\n",
     "        xarray.DataArray: The false-color image with the shortwave infrared, near-infrared, and red bands.\n",
     "    \"\"\"    \n",
-    "    image_path = image_dir.split(\"\\\\\")[-1]\n",
+    "    image_path = image_dir.split(dir_separator)[-1]\n",
     "    \n",
     "    if MASK_IMAGES:\n",
     "        image = [rioxarray.open_rasterio(f\"{image_dir}/{image_path}_B{x}.{ext}\").rio.clip(mask_geom, fire_events.crs) for x in [7,5,4]]\n",
@@ -297,17 +302,6 @@
     "    return NBR"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "66357e9e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "image_dir = f\"E:/Projects/UCB/FiredPy/data/Fire_events/{fire_id}/Landsat/\"\n",
-    "file_address = glob(os.path.join(image_dir, '*_T1'))"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 14,
@@ -340,6 +334,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "b5684035",
    "metadata": {},
@@ -542,15 +537,25 @@
     "fig.savefig(f'images/masked/Landsat/classified_dNBR_event_{fire_id}.png', dpi=300, bbox_inches='tight')"
    ]
   },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "faa6b451",
+   "metadata": {},
+   "source": [
+    "### Save Tiff files"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 20,
    "id": "74677997",
    "metadata": {},
    "outputs": [],
    "source": [
+    "# save pre and postfire NBRs as tiff files\n",
     "for image_dir, label in zip(file_address, labels):\n",
-    "    f_name = os.path.join(image_dir.split(\"\\\\\")[0], f'NBR_{label}.tif')\n",
+    "    f_name = os.path.join(image_dir.split(dir_separator)[0], f'NBR_{label}.tif')\n",
     "    nbr_raws[label].rio.to_raster(f_name)"
    ]
   },
@@ -561,7 +566,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "f_name_diff = os.path.join(image_dir.split(\"\\\\\")[0], f'dNBR pre-post fire.tif')\n",
+    "# save dNBR as tiff file\n",
+    "f_name_diff = os.path.join(image_dir.split(dir_separator)[0], f'dNBR pre-post fire.tif')\n",
     "dNBR_post_raw.rio.to_raster(f_name_diff)"
    ]
   },
@@ -582,11 +588,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Save pre and postfire RGB images as tiff files\n",
     "rgb_images = dict.fromkeys(labels, None)\n",
     "for image_dir, label in zip(file_address, labels):\n",
     "    rgb = open_RGB(image_dir, \"TIF\", False, False)\n",
     "    rgb_images[label] = rgb\n",
-    "    f_name = os.path.join(image_dir.split(\"\\\\\")[0], f'RGB_{label}.tif')\n",
+    "    f_name = os.path.join(image_dir.split(dir_separator)[0], f'RGB_{label}.tif')\n",
     "    rgb_images[label].rio.to_raster(f_name)"
    ]
   },
@@ -597,57 +604,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Save pre and postfire false color composite image as tiff\n",
     "fcc_images = dict.fromkeys(labels, None)\n",
     "\n",
     "for image_dir, label in zip(file_address, labels):\n",
     "    rgb = open_false_color(image_dir, \"TIF\", False, False)\n",
     "    fcc_images[label] = rgb\n",
-    "    f_name = os.path.join(image_dir.split(\"\\\\\")[0], f'false_color_{label}.tif')\n",
+    "    f_name = os.path.join(image_dir.split(dir_separator)[0], f'false_color_{label}.tif')\n",
     "    fcc_images[label].rio.to_raster(f_name)"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 17,
-   "id": "89b08c77",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "((23223, 16350), (23193, 16590))"
-      ]
-     },
-     "execution_count": 17,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "nir_pre.shape, nir_post.shape"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 18,
-   "id": "2fa3701c",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "((23223, 16350), (23193, 16590))"
-      ]
-     },
-     "execution_count": 18,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "swir_pre.shape, swir_post.shape"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 15,
@@ -716,22 +682,6 @@
     "# fig.savefig(f'images/masked/Landsat/NIR_and_SWIR_difference_event_{fire_id}.png', dpi=300)"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "769be370",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "230c2f51",
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
   {
    "cell_type": "code",
    "execution_count": 20,