Uncertainty estimates and bootstrapping

[tlienart]

Obtaining realistic uncertainty estimates is typically difficult but it'd be nice to offer some way of getting such estimates. I see effectively three main ones:

Approach	Assumptions	Scale to large n	Scale to large p	Difficulty	When
Full Bayesian	Many	Yes but not meaningful	Not in general	Hard	> medium term
Approx Bayesian	Many ++	Yes but not meaningful	Yes but not meaningful	Medium	medium term
Bootstrap	Few	Yes	Not in general	~Easy	short term

There may be other approaches or flavours of the above 3 (?, input welcome).

I think implementing/suppoorting bootstrapping soon-ish would be a good idea even though it's not perfect and does not work well in high dim AFAIK (though I don't know what does actually work in high dim...). There's already a package Bootstrap.jl and I'm not sure whether it's best to just interface with it via MLJModels or to write our own stuff.

Thoughts?

PS: to people who may be offended by the table above, apologies, it's not my intention to start a flame war; if you have suggestions and can guarantee that they offer realistic estimates in non-toy situations, please add them here.