osm_completeness

This series of notebooks uses a random forest regression model with google earth engine and python to predict OSM building mapping in afghanistan. notebooks can be adapted for use in any country. built by new light technologies for the world bank.

1. First, an Observable notebook is used to identify those areas of a given region that already have complete OSM building mapping. These areas are used as a training set for building the regression model, imported in step 2 (Train OSM)
2. TrainOSM.ipynb — (jupyter) create and train a random forest regression model that predicts OSM building footprint area using a variety of satellite imagery sources, including NTL, NDVI, NDBI, GUF, WorldPop, DEM, Landcover, etc. The output model from this notebook will be called in ApplyModel.ipynb.
3. SplitArea.ipynb — (jupyter) create 250-m polygon tessellations over your region of interest from a shapefile. This will allow your area of interest to be assessed for completeness.
4. Enrich.ipynb — (jupyter) enrich your tessellation AOI (from SplitArea.ipynb) with the data to be used for the model.
5. ApplyModel.ipynb — (jupyter) run the tessellation AOI through the trained model to calculate predicted OSM footprint area
6. Results of ApplyModel.ipynb can be visualized in a second Observable notebook. Be aware of a 15 MB limit to the file that can be attached to the notebook. For example, Kabul Province was divided into 4 pieces in order to be visualized.

contents:

• afghanistan - this folder contains notebooks used for the Afghanistan implementation of the OSM Completeness tool. Notebooks with '602' in the name used Jo's 602 training samples for Kabul and Jalalabad areas. Notebooks with '602_LC_DEM' in the name include substitution for a multiclass landcover dataset and higher resolution DEM data.
• envs - this folder contains the environment.yml file
• notebooks - this folder contains the original notebooks from Corey's Afghanistan implementation.

environment set-up:

The conda environment necessary to run the Jupyter notebooks can be installed using the file envs/wbenv.yml. To build it enter:
conda env create --name envname --file wbenv.yml
in the the anaconda terminal when inside the directory containing the yml file.

installation troubleshooting:

Users may encounter issues when installing the wbenv.yml in the anaconda terminal. Error message may be similar to one or both of the following:

'CondaVerificationError: The package for jedi located at [PATH] appears to be corrupted...'

AND/OR:

'ClobberError: This transaction has incompatible packages due to a shared path...'

The workaround for this is to do the following:

1. Remove the environment you created during set-up: conda -remove PATH.yml
2. Enter: conda -clean --all to remove index cache, lock files, unused cache packages, and tarballs.
3. Re-install the environment using: conda env create --name envname --file wbenv.yml

Windows machines may also encounter an issue with the rtree dependency for geopandas, specifically the error 'OSError: could not find or load spatialindex_c-64.dll' when importing geopandas. The workaround is to find the core.py file in the rtree library and change the line:

elif 'conda' in sys.version:

to

elif os.path.exists(os.path.join(sys.prefix, 'conda-meta')):

Then, deactivate and reactivate the environment.