Bootstrapping standard errors for Random or Causal Forests?

[kmoeltner]

Dear GRF community,
I understand that RF's and CF's can produce asymptotically valid standard errors and confidence intervals if certain rules are followed in their construction (e.g. honesty). I also understand that the (excellent!) GRF package generates standard errors as part of predictive output, if called.

My question: Could asymptotically valid s.e.'s also be obtained by bootstrapping the training sample, running the RF or CF (potentially with preceding residualization), generating predictions on the test sample (or a left-out observation from the training sample), and repeating many times? Then use the empirical standard deviation of the prediction as estimate for the standard error?

I'm asking because if bootstrapped s.e.'s are valid (= have the same desirable asymptotic properties as the ones generated by GRF), that would be very helpful in my current application, where I further process forest predictions using (highly nonlinear) functions.

Thank you very much for creating this user forum and for providing helpful feedback.

[swager]

The short answer is that the standard bootstrap (as you describe it) is valid, but with a caveat. The caveat is that random forests have monte carlo error from random sampling across trees, and this monte carlo error will inflate the standard bootstrap variance estimate. So this creates a tension: To get useful (short) intervals it's better to run each forest inside the bootstrap using very many trees -- but this will be slow (e.g., 2000 trees / forest x 500 bootstrap replicates = 1 million trees total).

What we do in GRF is to run a bootstrap-like algorithm, but with an analytic correction to eliminate the monte carlo bias. This procedure, which we refer to as the "bootstrap of little bags", is described in Section 6.1 of the GRF paper. We further embed this bootstrapping within our forest-growing algorithm to avoid any extra computational burden: While training the forest, we actually train groups of ci.group.size (default = 2) trees on each drawn half sample, and can then read off a bootstrap-like variance estimate by comapring within- versus between-group covariance of these trees.

Finally, one thing to note is that non-linear post-processing of the predictions may invalidate arguments used to justify the bootstrap here -- so that's also something to keep an eye out for.

[kmoeltner]

Hi Stefan, Thank you very much for your prompt reply - I will look at the ``little bags'' procedure one more time and give my strategy some more thought. *___________________________________________________________________________________* *Klaus Moeltner* 601 Jefferson St. Blacksburg, VA 24060 cell phone: (775) 560-3288 e-mail: ***@***.***

…

On Sat, May 3, 2025 at 3:59 PM Stefan Wager ***@***.***> wrote: *swager* left a comment (grf-labs/grf#1495) <#1495 (comment)> The short answer is that the standard bootstrap (as you describe it) is valid, but with a caveat. The caveat is that random forests have monte carlo error from random sampling across trees, and this monte carlo error will inflate the standard bootstrap variance estimate. So this creates a tension: To get useful (short) intervals it's better to run each forest inside the bootstrap using very many trees -- but this will be slow (e.g., 2000 trees / forest x 500 bootstrap replicates = 1 million trees total). What we do in GRF is to run a bootstrap-like algorithm, but with an analytic correction to eliminate the monte carlo bias. This procedure, which we refer to as the "bootstrap of little bags", is described in Section 6.1 of the GRF paper <https://arxiv.org/pdf/1610.01271>. We further embed this bootstrapping within our forest-growing algorithm to avoid any extra computational burden: While training the forest, we actually train groups of ci.group.size (default = 2) trees on each drawn half sample, and can then read off a bootstrap-like variance estimate by comapring within- versus between-group covariance of these trees. Finally, one thing to note is that non-linear post-processing of the predictions may invalidate arguments used to justify the bootstrap here -- so that's also something to keep an eye out for. — Reply to this email directly, view it on GitHub <#1495 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN2O5OXHIJABPMVZSG7G4N324UNZ5AVCNFSM6AAAAAB4L3CXASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBYG44DGOBSGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>