Reparametrization with exponential map instead of control variate

[ismailsenoz]

As discussed with @Nimrais, we might speed up projections if we utilize exponential map to reparameterize the gradient to reduce variance instead of control variate strategy.

[albertpod]

Just curious, speeding up by a factor of what?

Currently I'm struggling to run anything with CVIProjections that involves vectors which dimensionality exceeds 10.

Can @ismailsenoz or @Nimrais provide some example? Maybe if more context is supplied, some people will be more eager to tackle the issue?

[Nimrais]

Currently I'm struggling to run anything with CVIProjections that involves vectors which dimensionality exceeds 10.

Btw It would be nice if you could drop such an example as an issue here.

[ismailsenoz]

Just curious, speeding up by a factor of what?

Well I dont know what kind of a speed up we would gain. I think it might be an interesting research direction. Giving a definitive answer on the speed up factor would require a benchmarking and complexity analysis. I bet it would heavily depend on the specifics of the problem. It would be nice if you can supply the type of problem you are interested so people who are interested in can take it as a baseline for a comparison.

Currently I'm struggling to run anything with CVIProjections that involves vectors which dimensionality exceeds 10.

Perhaps the default settings of the projection are not allowing you to scale? You can try to decrease the number of samples to compute gradient estimates, for example. In any case, I do agree that projection mechanism is far from being optimal and not robust as it is and will not argue that it scales to the dimensions of the current SOTA algorithms you might have in mind.