Prediction grid covariates progress

[mitchellmanware]

• HMS
• NLCD
• GMTED
• GEOS
• NARR
• NARR 1-day lag
• MODIS MOD11
• MODIS MOD06
• MODIS MOD13
• MODIS MCD19 1km
• MODIS MCD19 5km
• MODIS MOD09
• MODIS VIIRS
• Koppen Geiger
• SEDAC Population
• TRI
• NEI
• Ecoregions
• SEDAC gRoads

[sigmafelix]

Are HMS targets still running? In my workstation, these targets were run in 1.76 days with 30 workers.

[mitchellmanware]

@sigmafelix
HMS and GMTED covariates have been calculated for the prediction grid, but we patterned by space and time so that each dynamic branch got a subset of dates and locations. Implementing the space only mapping had >2 day run times for some of the branches, and we did not run to completion.
Why do you think this performance would be so much worse for ours?

[sigmafelix]

@mitchellmanware
I was curious if it takes too long to run, which necessitates updates for efficiency. A possible reason for performance loss is at Linux kernel version in HPC (which may affect disk I/O speed) and/or SLURM. From what I tried, I could easily run spatial-only targets like GMTED, Koppen-Geiger, SEDAC, NEI in a few hours.

To note, GEOS-CF targets might take longer time than any other targets. NASA MODIS/VIIRS targets will run in a reasonable time if the raw files are preprocessed. Thank you for checking out the calculation targets!

[kyle-messier]

Thanks @sigmafelix - A couple notes from recent tests. We should probably use the par_grid_balanced from chopin to create the spatial grids. Currently the raw grid was creating a significant imbalance in the memory usage.

Second, we are testing and hopefully using sql commands with sqltargets and duckdb to subset the massive space-time grid. But we need lat/lon/time coordinates, which the raw output of the dynamic branches for the grid coordinates do not have. So I think we need a chopin target that retains this information for easier querying later.

[sigmafelix]

@kyle-messier Sure, I have seen the imbalance and its impact to memory usage either. sf_pred_calc_grid has 8092995 rows, and its divisors are 1, 3, 5, 15, 539533, 1618599, 2697665, and 8092995 (I was surprised to see that 535933 is a prime number), which limit the utility of par_grid_balanced() because anticlust::balanced_clustering() in the background requires the exact number of clusters that is a divisor of the total number of rows. To reduce memory usage, we can increase the number of grids or modify the computation functions to chunk the input object into manageably-sized ones (e.g., 10000). duckdb approach looks promising. Do you plan to use SQL directly in each target or use dbplyr?

[mitchellmanware]

@sigmafelix For the HMS prediction grid calculation, is this the code which ran in 1.76 days? I see you changed from lapply method to for loop.

 results <- vector("list", length(list_dates))
        
        for (i in seq_along(list_dates)) {
          d <- list_dates[[i]]
          
          results[[i]] <- beethoven::inject_calculate(
            covariate = "hms",
            locs = list_pred_calc_grid,
            injection = list(
              path = file.path(chr_input_dir, "hms", "data_files"),
              date = beethoven::fl_dates(unlist(d)),
              covariate = "hms"
            )
          )[[1]] |>
            dplyr::select(-dplyr::any_of(c("lon", "lat", "geometry", "hms_year")))
          
          # Force garbage collection every 10 iterations
          if (i %% 10 == 0) {
            gc()
          }
        }
        
        # Use data.table for more efficient binding
        final_result <- data.table::rbindlist(results, use.names = TRUE, fill = TRUE) |>
          as.data.frame()
        final_result

beethoven/inst/targets/targets_calculate_predict.R

Line 123 in cb9e584

########################### HMS ###########################

[sigmafelix]

@mitchellmanware Yes, I switched to for-loop as this approach used less memory. This choice comes from the memory constraints in my workstation. Do you think the modification needs to be merged into main?

[kyle-messier]

Updating to do prediction calculations on H3 Resolution 8 locations

• HMS
• NLCD
• GMTED
• GEOS
• NARR
• NARR 1-day lag
• MODIS MOD11
• MODIS MOD06
• MODIS MOD13
• MODIS MCD19 1km
• MODIS MCD19 5km
• MODIS MOD09
• MODIS VIIRS
• Koppen Geiger
• SEDAC Population
• TRI
• NEI
• Ecoregions
• SEDAC gRoads